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Managing software development in large organizations has become increasingly difficult 
due to constantly increasing technical complexity, stricter government standards, a 
shortage of experienced software engineers, competitive pressure for improved 
productivity and quality, the need to co-develop hardware and software together, and he 
rapid changes in both hardware and software technology. 

The "software factory' approach to software development minimizes risks while 
maximizing productivity and quality through st a ndardization, automation, and training. 
However, in practice, this approach is relatively inflexible when adopting new software 
technologies. How can a lage multi-project software engineering organization increase 
the likelihood of successful software technology insertion (STI), especially in a 
standardized engineering environment? 

HISTOGRAM of SOFTWARE TECHNOLOGY INSERTION CASES 
8 

6 

4 

2 

0 

-12 -6 0 +6 +12 
Failure < ■ - ■» Success 

Rgure 1 - Distribution of scores from 49 ra ted STI Cases 

In an attempt to correlate various success factors with levels of success, 59 cases of "new 
software technology insertion* in thirteen recent projects at a large U.S. Defense 
electronics contractor were identified and categorized according to several criteria. The 
relative success or failure of 49 of these cases (see Rgure 1) was determined by 
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having key project personnel (Lead Engineer, Dept Manager, and tool supporters) rate 6 
aspects (added together for total rating) of the software technology insertion results. 
Maximum success was scored as +12, and maximum failure as -12 on the rating scale. 
The histogram in Figure 1 illustrates the dstribution of scores from the 49 rated cases. 

There were 21 Afferent new software technologies studied, most of them new toots 
or methods, inducting (in approximate lifecyde order): 

• The use of DoD-STD-2167 or 2167A 

• Structured analysis CASE tools 

• Rapid-Prototyping in requirements or design 

• In-House requirements traceability tool 

• In-House program design language (POL) tools 

• Reusable Software in design or coding 

• The use of Ada® as an implementation language 

• The use of M68C2Q assembly language 

• Microprocessor Development Stations (MDS) for integration testing 

• In-House configuration management (CM), source code control tool 

• Workstation-based engineering documentation tools 

• The use of workstations as primary development platforms 

Though meaningful statistical correlations were not possible due to the limited sample 
size, ratings were compiled and empirically compared with several technology factors 
measured for each ST1 case, induding: 

• Technology Type (Competence-Enhandng or -Destroying) 

• Support Type (In-House or External) 

• Maturity of the Technology (Young, Mature, and Old) 

• Project Size (SLOC) 

• Prior Expectations (for success or failure) 

• Reasons (for using the new software technology) 

• Methods (of inserting the new software technology) 

• Perceived Time Savings 

• Perceived Labor Savings 

• Perceived Computer Cost Savings 

• Perceived Quality Improvement 

• Met Expectations? (for success or failure) 

A doser look at the Top Eleven* cases of successful ST1 (ratings 2 +7), and the 
‘Bottom Seven* cases of unsuccessful ST1 (ratings £ -7) shows that: 

1 . Perceived Time Savings and perceived Labor Savings are the 
most significant real indicators of successful or unsuccessful STL 

2. Though users often complain about increased computer costs, 
saving computer cost is not an indicator of STI success, because it 
is not usually a goal or a motivator for the use of new technology. 

3. Perceived Quality Improvement is a strong indcator of STI 
success, but not an indcator of STI failure. 
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4. Even in successful STI cases, users' Prior Expectations about 
what a new technology can/cannot do are not managed effectively. 

In addition to the success ratings, on-site structured Interviews were used to profile 
each new technology, and collect other qualitative information that was used to clarify 
and complete the data. 

Tushmanfl] describes new technology types as: (1) competence-enhancing - 
incrementally cfifferent, buiidng on existing know-how, and substituting for older 
technologies without rendering their skills obsolete, or (2) competence-destroying • 
fundamentally cfifferent requiring new skills, abilities, and knowledge for use. The main 
types of technology support are: (1 ) In-House, where the supporters work in the same 
organization as the users, or (2) Outside, where the supporters work in a different 
organization than the users. 

A sample of the dstribution of successful STI cases over these two combined factors 
(technology type and support type) is show in Figure 2: 

Ratings of Ne w Technology Types Across Two Dimensions 


IN-HOUSE OUTSIDE 

Support Support 


Competence 

ENHANCING 

♦Total- 1 6 

♦Rated- 1 3 

Tot Rating- 47.0 
Median- +5.0., — 
Ave Rating- (+3.6) 

♦Total- 9 

#Rated- 8 

Tot Rating- 11.5 
Median- +4.0 , — . 
Ave Rating- (+1 . 4 ) 


BEST 

OK 

Competence 

DESTROYING 

♦Total- 1 1 

♦Rated- 8 

Tot Rating- -2.0 
Median- +1.5.. — 
Ave Rating- 

♦Total- 23 

♦Rated- 20 

Tot Rating- 14.5 

Median- +07 ^ 

Ave Rating- (+0/y 


Poor 

Marginal 


RATING SCALE: +12 - Maximum Success, -12 « Maximum Failure 
Figure 2 - Distributio n of Success/Failure across two factors 
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The new software technologies that had the most successful ST! experience (though 
across a very limited set of cases) are summarized below: 


as 


1 

2 
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Ave Rating NflW Software TetflDQlOOY 
+9.5 In-House Automated Build Tool 

+7.5 Microprocessor Development Stations for Integration 
+4.8 In-House Software Problem Reporting Tool 
+3.3 In-House Configuration Maiagement (CM) Tod 

+2.1 In-House Program Design Language Tool 


The new software technologies that had the least successful ST! experience are: 


#Cases Ave Rating New Software, Treftnology 

1 -110 In-House Automated Code Documentation Tod 

2 -8.8 Workstation-based Engineering Documentation Tool 

4 -4.8 Workstation-based CASE Tool for Reqts and Design 


Among the overall conclusions from the study are: 

1 . Saving schedule time and labor costs are necessary and sufficient 
conditions for successful ST! 

2. Improving quality is a necessary, but not suffident condition for 
successful Software Technology Insertion (ST!) 

3. Success with new software technology insertion (ST1) is much greater for 
competence-enhancing than for competence-destroying technologies 

4. Success with ST1 is somewhat greater for in-house supported 
technologies than for outside supported technologies 

5. Success with ST! is greater for mature technologies than for either young 
or old technologies (mature is >1 year after release, <5 yeas after release) 

6. Success with ST! is greater when users’ expectations about 'new 
technology' are controlled to avoid expecting too much - exceecfing users' 
expectations is not necessary for successful ST1, but qq! meeting 
expections (i.e., dsappointing them) is a suffident condition for failure 

7. Success with ST! can be increased when there is synergy between 
multiple new technologies, such as Ada and workstations 


These and other results and condusions, along with some recommendations fcr large 
software development organizations, will be covered at the workshop. 
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13 Software Projects 


Project ID* 

Language 

SLOC 

Current Pheae 

1 

Assembly 

492 OC 

Integration Test 


Fortran 

6400 



C 

2300 


2 

C 

9100 

Desion & Code 

3 

Assembly 

4000 

Maintenance 


C 

4500 


4 

C 

8500 

lrrtea ration Test 

5 

Assembly 

1085C 

Desiqn & Code 

6 

Assembly 

7100 

System Test 

7 

Assembly 

1600C 

Intea ration Test 

8 

Fortran 

n/a 

Cancelled 

9 

Ada 

2500C 

Deskjn 

10 

Ada 

n/a 

lrrtea rati on Test 

11 

Assembly 

4850 

lrrtea rati on Test 1 

12 

C 

1370C 

Maintenance 

13 

Assembly 

1800C 

Design & Code 


21 New Software Technologies 

(most of them new tools, methods, languages) 

A - The use of Ada® as an implementation language 
B - In-House automated build tool(s) 

C - In-House automated code documentation tooi 

D - In-House program design language (PDL) tools 

E - In-House metrics tools for automatic data collection 

G - In-House standard test reporting tool based on RDBMS 

I - Workstation-based engineering documentation tool 

J - The use of M68020 assembly language 

K - Microprocessor Development Stations (MDS) for integration testing 

L - In-House project scheduling and reporting tool 

M - In-House configuration management (CM), source code control tool 

N - In-House Vax/Unix documentation package using troff 

P - In-House Software Problem Reporting Tool based on RDBMS 

R - Rapid-Prototyping in requirements or design 

S - Structured analysis graphical CASE tool 

T - Structured analysis graphical CASE tool 

U - Reusable Software in design or coting 

W - The use of workstations as primary development platforms 

X - Workstation-based engir Bering documentation tool 

Y - In-House requirements traceability tool 

Z - The use of DoD-STD-21 67 or 21 67A 
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Project/Technology Matrix 

New Technology ID 
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59 ST1 Cases: % Rated Q Not Rated 


Measuring Perceived ST1 Success 

• For each STI Case, 6 Questions were asked of: 

(1) Lead Engineer (Project/Matrix) 

(2) Dept Manager (FunctionaJ/Matrix) 


For Each STI Case: 

Statement (Agree or Disagree?) 

Agree — Disagree 
+2 +1 0 -1 -2 

1. 1 would use the new method/tool again 

2. The new method/tool saved schedule time 

3. The new method/tool saved labor cost 

4. The new method/tool saved computer cost 

5. The new method/tool Improved quality 

6. The new method/tool met my expectations 

V 

V 

V 

V 

>1 


* Total Rating for each STI Case is sum (example =+1) 
i.e., maximum = +12, minimum = -12 


(Note: Questions not weighted) 
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Software Technology Insertion (STI) 


Software Technology Insertion 
"New" Software Technology 

Opportunity to Insert 


"New" Software Technology 
+ Opportunity to Insert 

Tool or Method that is unfamiliar 
to the majority of a Project Team, 
usually replacing a more familiar one 

A software development activity on 
a new (most likely) or ongoing (less 
likely) software project 
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Successful STI 


Perceived STI Success User's sense of Labor Cost Savings + 

User's sense of Computer Cost Savings + 
User's sense of Elapsed Time Savings + 
User's sense of Quality Improvement 

Real STI Success Measured Labor Cost Savings + 

Measured Computer Cost Savings + 
Measured Elapsed Time Savings + 
Measured Quality Improvement 
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STI "Cases" Overview 


STI Case 

59 STI Cases Identified 

49 STI Cases Rated for 
Perceived Success 

13 Different SW Projects 
p p 21 Different SW Technologies 

is 


A single incident of STI 
on a single project, usually 
within a single development phase 

Across 13 different projects; 
from 1 to 7 STI Cases per project 

Some of the 59 identified cases 
were not able to be rated 

Some ongoing, some just completed; 
using Ada, C, Fortran, Assembly; 
ranging in size from 2900 to 49200 SLOC 

Most new tools, methods, languages (e.g., 
CASE, 2167A, Ada, Rapid-Proto, Reuse,...) 
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13 Software Projects 


Project ID# 

Language 

SLOC 

Current Phase 

1 

Assembly 

49200 

Integration Test 


Fortran 

6400 



C 

2900 


2 

C 

9100 

Desian ft Coda 

3 

Assembly 

4000 

Maintenance 


C 

4500 


i 4 

C 

8500 

Intearation Test 

5 

Assembly 

10850 

Desian & Code 

6 

Assembly 

7100 

System Test 

7 

Assembly 

16000 

Intearation Test 

8 

Fortran 

n/a 

Cancelled 

8 

Ada 

25000 

Desian 

10 

Ada 

n/a 

Intearation Test 

11 

Assembly 

4850 

Intearation Test 

12 

C 

13700 

Maintenance 

13 

Assembly 

18000 

Design & Code 
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21 New Software Technologies 

(most of them new tools, methods, languages) 

A - The use of Ada® as an Implementation language 
B - In-House automated build tool(s) 

C • In-House automated code documentation tool 
D - In-House program design language (PDL) tools 
E - In-House metrics tools for automatic data collection 
Q - In-House standard test reporting tool based on RDBMS 
I - Workstation-based engineering documentation tool 
J • The use of M68020 assembly language 
K - Microprocessor Development Stations (MDS) for integration testing 
L - In-House project scheduling and reporting tool 
M - In-House configuration management (CM), source code control tool 
N - In-House Vax/Unix documentation package using troff 
P - In-House Software Problem Reporting Tool based on RDBMS 
R • Rapid-Prototyping in requirements or design 
S - Structured analysis graphical CASE tool 
T - Structured analysis graphical CASE tool 
U - Reusable Software in design or coding 
W • The use of workstations as primary development platforms 
X * Workstation-based engineering documentation tool 
P J! Y - In-House requirements traceability tool 

1 1 Z * The use of DoD-STD-2167 or 21 67 A 
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New Technology ID 

ABCDEG I JKLMNPRSTUWXYZ 
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99 8TI Cases: 


Rated O Not Rated 
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Other Measured Factors 


• Technology Type (Competence-Enhancing or -Destroying) 

• Support Type (In-House or External) 

• Maturity of the Technology (Young, Mature, and Old) 

• Project Sire (SLOC) 

• Prior Expectations (for success or failure) 

• Reasons (for STI choice) 

• Methods (of STI insertion) 

• Perception of Time Savings 

• Perception of Labor Savings 

• Perception of Computer Cost Savings 

• Perception of Quality Improvement 

• Result vs. Prior Expectations (for success or failure) 
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Measui ing Perceived STI Success 


• For each STI Case, 6 Questions were asked of: 

(1) Lead Engineer (Project/Matrix) 

(2) Dept Manager (Functional/Matrix) 


For Each STI Caaa: 

Statement fAaree or Disaareo?) 

Agree Disagree 

♦2 +1 0 -1 -2 

1. 1 would use the new method/tool again 

__i 

2. The new method/tool saved schedule time 

JL 

3. The new method/tool saved labor cost 

V 

4. The new method/tool saved computer cost 

V 

5. The new method/tool Improved quality 

i 

6. The new method/tool met my expectations 

_4_ 


• Total Rating for each STI Case is sum (example =+1) 
i.e., maximum = +12, minimum = -12 
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(Note: Questions not weighted) 
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Histogram Of Software Technology Insertion Cases 
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Top Eleven STI Cases 

Main reasons for "success": 

• "Synergy" wtthin a project (4) 

• Critical need for a capability (3) 

• "Synergy" between two technologies (2) 

• Mature and powerful tool (1) 

May or may not "Save Computer Costs" (+0.2) 

"Met Expectations" (+0.5) not as critical as: 

• "Save Time’ (+1.8) 

• "Save Labor" (+1.7) 

• "Improve Quality" (+1.6) 


Failure -«*• 


Success 
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" Bottom Seven " I I 

4 - I (Rating £ -7) j j-j 

| || || |i 

fi i i inn ii 

III 1 1 1 II HII 1 1 1 1 II 1 


Main reasons for "failure": 

• Immature technology (3) 

• Interface problems (3) 

• Technology not “needed" by LE (2) 

• Wrong technical solution (1) 

May or may not "Improve Quality" (-0.4) 

"Save Computer Costs" (-1.1) not as critical as: 

• “Save Time* (-1 .8) 

• “Save Labor - (-1 .9) 

• “Met Expectations" (-1.9) 


Failure 


Success 
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Competence-Enhancing vs Competence-Destroying 

Competence-Enhancing technology - major improvement in price/performance 
that builds on existing know-how; a substitute for older technology, but does 
not render old skills obsolete; increase efficiency. 

Competence-Destroying technology - new way of making a given product; 
requires new skills, abilities, and knowledge for use; may combine previously 
discrete steps into continuous flow, or be a completely different process 

Maturity of a New Software Technology 

Young technology - Released < 1 year, or prior to 2nd major release (VI. x) 

Mature technology - Released > 1 year, and after 2nd major release (V2.x+) 

| £ Old technology - Released > 5 years, or after end of formal support 
5 s 
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Ratings of New Technology Types Across Two Factors 



IN-HOUSE Support OUTSIDE Support 


Competence 

ENHANCING 

("incremental") 

#Total= 16 

#Rated- 13 

Tot Rating= 47.0 
Median- +5.0^- — 
Mean Rating= (+3.6) 

#Total= 9 

# Rated- 8 

Tot Rating- 11.5 
Median- +4.0 — v. 
Mean Rating- (+1.4J 


BEST 

OK 

Competence 

DESTROYING 

("radical") 

#Total- 1 1 

#Rated= 6 

Tot Rating- -2.0 
Median- +1.5^ — 
Mean Rating- Qo.2^ 

#Total= 23 

#Rated- 20 

Tot Rating- 14.5 
Median- +0.7 — >. 
Mean Rating- (*0.7) 


Poor 

Marginal 


Mean = +2.1 

Mean = +0.9 


Mean s +2.8 


Mean & +0.4 


RATING SCALE: +12 « Maximum Success, -12 - Maximum Failure 
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Summary of Results 

(Focus on success factors rather than successful technologies) 

(Focus on perceived rather than real STI success) 

• Saving schedule time and labor costs drive successful STI (obvious?) 

• Improving quality is necessary, but not sufficient for successful STI 

• Exceeding users' expectations not necessary for successful STI, but 
not meeting expectations is sufficient for failure (i.e., must control) 

• Much greater success for competence-enhancing 
vs competence-destroying technologies 

• Greater success for mature vs young or old technologies 

| £ • Somewhat greater success for in-house vs outside supported 

Is 
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Next Step: 

Linking Perceived Success with Real Success 
via Software Metrics Collection 


• Corporate-wide effort to implemer n automatic collection of software 
metrics as a by-product of development - MSD is Lead Division 

•10 current software metrics defined (similar to Mitre Metrics) 

• Based mainly on DoD-STD-2167A 

• AutoCollection in development for both project-specific and 
process-level (across multiple projects) software metrics 
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Overview of Raytheon MSD's Software Metrics Collection 
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